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This  effort,  which  was  sponsored  by  NAVPERSRANDCEN  management,  is  a  continua¬ 
tion  of  a  FY81  reimbursable  work  unit  (Implementation  Planning)  sponsored  by  the  Chief 
of  Naval  Operations  (OP- 1 1 5).  Its  objective  was  to  expand  and  refine  the  product  tracking 
system  that  was  developed  in  FY81  and  described  in  NPRDC  SR  82-29. 

The  overall  findings  indicate  that  a  user-oriented  tracking  system  can  provide  the 
means  for  initiating  and  maintaining  a  productive  dialogue  between  operational  consumers 
and  research  activities. 
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INTRODUCTION 


Problem  and  Background 

The  concern  of  the  Congress  and  the  Department  of  Defense  (DoD)  about  the 
relevance  and  use  of  human  resource  RDT&E  end  products  has  increased  during  the  past 
decade.  This  concern  has  emphasized  a  need  to  develop  methods  to  increase  the 
probability  that  research  products  and  outcomes  will  be  used.  To  illustrate,  in  1977,  the 
General  Accounting  Office  (GAO)  recommended  that  a  management  monitoring  and 
feedback  system  on  research  utilization  be  developed  and  implemented.1 

In  response  to  this  concern,  the  Navy  Personnel  Research  and  Development  Center 
(NAVPERSRANDCEN)  designed,  implemented,  and  tested  a  system  to  track  and  evaluate 
application/ utilization  of  Center  RicD  products.2  During  FY81,  this  system  was  pilot- 
tested,  using  a  selected  sample  of  NAVPERSRANDCEN  technical  reports,  special  reports, 
and  technical  notes  describing  Center  products.  Commands  previously  identified  as  user 
organizations  were  asked  to  review  the  reports  selected  and  complete  a  questionnaire 
addressing  the  following  areas: 

1.  Type  and  frequency  of  communications  between  the  user  and  NAVPERSRAND¬ 
CEN. 

2.  The  extent  of  user  involvement  in  phases  of  research,  including  planning, 
research  design,  analyses,  and  interim  or  final  recommendations. 

3.  Whether  or  not  the  user  conducted  a  formal  management  review  of  the  product. 

4.  Degree  of  satisfaction  with  the  product's  timeliness,  completeness,  clarity, 
relevance,  feasibility  of  implementation,  cost  of  implementation,  and  projected  benefits 
of  implementation. 

5.  Impact  of  the  research. 

6.  Overall  user  evaluation  of  the  research. 

Objective 

The  objective  of  this  effort  was  to  expand  and  refine  the  product  tracking  system 
developed  and  pilot- tested  in  FY81. 


1  Hum  an  resources  research  and  development  results  can  be  better  managed— Depart¬ 
ment  oi  ftelense.  ^Washington,  DC:  Comptroller  general's  report  to  the  Committee  on 
Appropriations,  House  of  Representatives,  April  22,  1977. 

*Rosen,  H.  H.  A  system  for  assessing  user  response  to  NAVPERSRANDCEN  RDT&E 
products  (NPRDC  Spec.  Rep.  82-29).  San  biego:  Navy  Personnel  Research  and 
Development  Center,  June  1982.  (AD-A 1 17  719) 


APPROACH 


During  FY82,  the  format  of  the  questionnaire  was  upgraded  (it  was  "professionally 
printed"  as  opposed  to  being  xeroxed)  to  attract  the  interest  and  attention  of  the 
recipients.  An  additional  section  was  added  regarding  the  user's  need  for  assistance  from 
NAVPERSRANDCEN  in  the  operational  implementation  phase  of  the  end  products  (see 
item  P4,  p.  A-2). 

In  FY82,  NAVPERSRANDCEN  published  127  reports.  User  evaluations  were  not 
requested  for  48  of  these  reports,  either  at  the  request  of  the  research  program  director 
or  because  the  reports  were  considered  inappropriate  for  evaluation  (e.g.,  bibliographies). 
For  the  remaining  79  reports,  160  evaluation  requests  were  sent  to  individuals  represent¬ 
ing  the  following  organizations: 

•  Chief  of  Naval  Education  and  Training  (CNET). 

•  Chief  of  Naval  Operations  (CNO). 

•  Chief  of  Naval  Material  (CNM). 

•  Chief  of  Naval  Technical  Training  (CNTT). 

•  Naval  Military  Personnel  Command  (NMPC). 

•  Navy  Recruiting  Command  (NRC). 

•  Headquarters,  Marine  Corps  (HQMC). 

•  Commander  in  Chief,  U.S.  Pacific  Fleet  (CINCPAC). 

•  Commander  in  Chief,  U.S.  Atlantic  Fleet  (CINCLANT). 

•  Naval  Air  Systems  Command, 

•  Department  of  Defense. 

•  U.S.  Naval  Academy. 

•  Center  for  Naval  Analyses. 

•  Naval  Surface  Weapons  Center. 

•  Commander  Naval  Air  Force,  U.S.  Pacific  Fleet. 

•  Commander  Naval  Air  Force,  U.S.  Atlantic  Fleet. 

•  U.S.  Coast  Guard. 


RESULTS 

A  total  of  119  evaluation  reports  were  returned,  for  a  response  rate  of  74  percent. 
Evaluations  were  not  returned  for  nine  of  the  reports. 

Thirty-eight  of  the  reports  returned  indicated  that  the  user  would  require  assistance 
from  NAVPERSRANDCEN  during  the  operational  implementation  phase.  In  these  cases, 
the  principal  investigators  were  alerted  for  appropriate  action. 

Analysis  of  evaluation  reports  showed  the  following: 

1.  Eighty-two  percent  of  the  respondents  believed  that  the  amount  and  kind  of 
communications  with  NAVPERSRANDCEN  during  the  course  of  the  research  were 
sufficient. 

2.  Fifty-one  percent  stated  they  were  actually  personally  involved  with  some 
aspect  of  the  research  effort;  and  78  percent,  that  the  degree  of  involvement  was 
sufficient  (i.e.,  they  responded  1,  2,  or  3  on  scale  of  6  to  item  IV,  P6,  p.  A-3). 


3.  Seventy-six  percent  stated  that  the  end  product  would  result  in  either  cost 
savings  or  increased  efficiency  or  effectiveness. 

4.  Thirty-nine  percent  indicated  that  the  product's  implementation  potential  had 
already  been  formally  reviewed. 

Users  were  asked  to  indicate  how  satisfied  they  were  with  various  aspects  of  the 
research  (Item  V,  p.  A-4).  Results,  which  are  presented  in  Table  1,  show  that  there  is  a 
differential  satisfaction  with  these  aspects  on  the  part  of  users. 

Table  1 

User  Satisfaction  With  Various  Aspects  of  NAVPERSRANDCEN  Research 


Very  or 

Neither  Satis- 

Somewhat  or 

Somewhat 

fied  Nor 

Very  Dis- 

N 

Aspect 

Satisfied 

Dissatisfied 

satisfied 

Responses 

(%) 

{%) 

(%) 

Relevance  of  findings 

to  the  problem 

77 

14 

9 

112 

Completeness  of  study 
Clarity  of  recommenda- 

77 

14 

9 

112 

tions 

Projected  benefits  of 

80 

14 

6 

112 

implementation 

71 

21 

8 

102 

Timeliness  of  response 

71 

21 

8 

111 

Feasibility  of  implemen- 

tation 

58 

31 

11 

103 

Cost  of  implementation 

55 

42 

3 

93 

A  global  estimate  of  user  satisfaction  obtained  showed  that  95  percent  of  all  users 

responding  were  satisfied  to  some  extent  with  the  research  product. 

Table  2  provides  a 

breakout  by  user. 

Table  2 

Overall  Satisfaction  With  NAVPERSRANDCEN  Research  by  User 

Percent 

Percent 

User 

Satisfied 

Dissatisfied 

CNET 

92 

8 

CNO 

100 

0 

CNM 

100 

0 

CNTT 

100 

0 

NMPC 

89 

11 

NRC 

100 

0 

HQMC 

100 

0 

C1NCPAC 

100 

0 

All  others 

88 

12 

3 


The  questionnaire  also  included  a  number  of  open-ended  questions  to  obtain  user 
comments  on  various  aspects  of  Center  research.  The  information  provided  by  such 
comments  is  extremely  valuable  as  a  diagnostic  aid,  substantially  increasing  the  quality  of 
feedback  to  Center  researchers.  The  followin'  comments  are  representative  of  those 
extracted  from  user  evaluations: 

1.  This  report  confirms  what  we  have  known  for  some  time:  The  Correctional 
Custody  program  is  effective  and  extremely  beneficial  to  the  Navy. 

2.  I  am  responsible  for  actions  taken  and  agree  with  the  findings  and  the 
recommendations  and  we  are  following  them. 

3.  ...  use  of  the  preliminary  objectives  has  already  proven  to  be  beneficial  to 
team  performance  of  ship  crews. 

4.  We  communicated,  but  there  was  not  always  listening. 

5.  No  worthwhile  recommendations  were  made. 

6.  Never  quite  sure  of  purpose  of  study. 

7.  Study  provides  excellent  background  on  role  of  instructors  in  self-paced  courses. 

8.  Results  appear  to  be  useful  and  applicable. 

9.  Good  comprehensive  piece  of  work. 

10.  The  results  of  this  study  have  caused  policy  makers  for  the  surface  community  to 
clearly  focus  on  our  major  problem  areas  which  were  not  previously  known  to  us. 

11.  Product  is  implemented  and  in  production  status. 

Most  of  the  comments  were  positive  and  reinforcing.  However,  comments  4,  5,  and  6 
indicate  that  some  users  feel  that  reports  do  not  respond  to  their  needs. 

Table  3,  which  provides  comparison  data  for  the  FY81  and  FY82  tracking  systems, 
shows  the  following: 

1.  Although  more  evaluation  requests  were  sent  and  received  in  FY82  than  in  FY81 
(160  and  119  vs.  133  and  99),  the  response  rate  (74  percent)  was  the  same  for  both  years. 

2.  In  FY82,  82  percent  of  the  users  felt  that  the  amount  and  kind  of  communication 
were  sufficient,  compared  to  79  percent  in  FY8I. 

3.  Only  51  and  64  percent  of  FY82  and  FY81  users  respectively  claimed  actual 
involvement  in  the  research.  However,  for  both  years,  78  percent  felt  the  degree  of 
involvement  was  sufficient. 

4.  In  FY82,  76  percent  of  users  predicted  that  the  product  would  result  in  either 
cost  savings  or  increased  efficiency  or  effectiveness,  compared  to  59  percent  in  FY81. 
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Table  3 


FY81/FY82  Tracking  System  Comparison  Data 


Item 

FY81 

FY82 

Number  of  reports  evaluation  requests  were  sent  for 

61 

79 

Number  of  evaluation  requests  sent 

133 

160 

Number  of  evaluation  requests  returned 

99 

119 

Response  rate 

74% 

74% 

User  felt  the  amount  and  kind  of  communication  with 
NAVPERSRANDCEN  were  sufficient 

79% 

82% 

User  claimed  actual  involvement  with  some  aspect  of  research 

64% 

51% 

User  felt  degree  of  involvement  was  sufficient 

78% 

78% 

User  predicted  that  the  product  would  result  in  either  cost 
savings  or  increased  efficiency  or  effectiveness 

59% 

76% 

The  most  significant  improvement  in  FY82  was  in  the  globa  *>mate  of  user 
satisfaction.  The  percentage  indicating  "moderate,"  "great,"  or  "vt  ;  .at"  extent  of 
satisfaction  increased  from  76  percent  in  FY81  to  87  percent  in  FY82. 


CONCLUSIONS 

The  FY82  data  indicated  that  communications  with  the  user  community  were  highly 
satisfactory  and  that  overall  satisfaction  with  the  Center's  end  products  increased  over 
that  reflected  in  FY81.  A  significant  number  of  users  predicted  that  a  product  would 
result  in  either  cost  savings  or  increased  efficiency  or  effectiveness. 


5 


APPENDIX 

RESEARCH  UTILIZATION  EVALUATION  REPORT 


A-0 


NAVY  PERSONNEL  RESEARCH  AND  DEVELOPMENT  CENTER  (NPRDC) 
RESEARCH  UTILIZATION  EVALUATION  REPORT 


Return  this  form  to  NPRDC  (Code  303),  San  Diego,  CA  92152 

(D8) 

(R1) 

(RS)  PorS 
(US) 

I.  SO  THAT  WE  CAN  PROVIDE  MORE  USEFUL  FEEDBACK  TO  OUR  RESEARCHERS  AND  RAD 

MANAGERS,  WE  ENCOURAGE  YOU  TO  EVALUATE  THE  RESEARCH  EFFORT  REPRESENTED 
itv- 


iTT) 


_ _ _ _ _ (PN) _ 

The  principal  investigator  for  this  project  is  (PI) _ 

_ _  .  ,  .  (CO) _  ,  (IV) _ 

II.  COMPLETED  BY  USER/SPONSOR: 

(UN)  Name: _ _ 

Last  Pint  Ml 

(RK)  Rank  or  Grade: _ 

(Tl)  Position:  _ 

(UV!  Autovon  Number: _ 

(UC)  Commercial  Phone  Number: _ 

(OS)  Office  Symbol: _ 


"Sase'or  Chy - 

State 

Zip 

(PO! 

User/Sponsor's  Parent  Organization: 

(TU) 

Type  of  organization:  1  Navy 

2  Armv 

3  Air  Force 

4  Marine  Cores 

_ 5  University  _ 

_ 6  Other 

NOTE:  The  NPRDC  Utilization  and  Appraisal  Office  is  responsible  for  this  evaluation  form.  If  you  have  any  questions, 
contact  NPRDC  (Code  303),  San  Diego,  CA  92152.  A/V:  933-7460  or  Commercial:  (714)  225-7460. 


*  * 


IMPORTANT  *  * 


P4) 


IF  YOU  WILL  BE  THE  USER  OF  THIS  RESEARCH-PRODUCT  -  OR  IF  YOU  HAVE 
A  SIGNIFICANT  RESPONSIBILITY  FOR  IMPLEMENTING  THE  PRODUCT- WILL 
YOUR  ORGANIZATION  REQUIRE  FOLLOW  UP  ASSISTANCE  FROM  NPRDC  IN 
THE  OPERATIONAL  IMPLEMENTATION  PHASE;  EVEN  THOUGH  THE  RESEARCH 
MAY  HAVE  BEEN  COMPLETED? 

YES? _ NO? _ UNKNOWN  AT  THIS  TIME _ 


III.  DURING  THE  COURSE  OF  THIS  RESEARCH  PROJECT,  ABOUT  HOW  OFTEN  WERE  EACH 
OF  THE  FOLLOWING  KINDS  OF  RESEARCHER-USER  COMMUNICATIONS  USED: 

(Fill  in  the  blank  with  one  of  the  following  letters  to  indicate  frequency ) 


a. 

Weekly  or  more  often 

d. 

Twice  a  year 

b. 

Monthly 

e. 

Once  a  year  or  less 

c. 

Every  two  months 

f. 

Never 

■  I C  i  _ Telephone  calls? 

1  *’r  ‘ _ Written  communications  (memos,  progress  reports,  etc.)? 

f’v  _ Personal  Visits? 

<CW? _ Conferences,  workshops? 

Indicate  your  estimate  of  the  research-user  communication  for  this  project  by  circling  the 
appropriate  scale  value  below 


I  2  3  4  S  6 


;  t"j  i  The  overall  amount 
was  sufficient 


There  should  have 
been  more 


A-2 


<» 


?  F y  Further  comment? 


IV.  TO  WHAT  EXTENT  WAS  YOUR  ORGANIZATION  INVOLVED  IN  THE  FOLLOWING  STAGES 

OF  THIS  RESEARCH  PROJECT: 


1. 

To  a  very  great  extent 

4. 

To  a  little  extent 

2. 

To  a  great  extent 

5. 

To  a  very  little  extent 

3. 

To  a  moderate  extent 

6. 

None 

7. 

Don't  know 

SPS)  _  Planning:  Identification  of  research  needs,  defining  the  objective(s),  scope,  data 

requirements,  etc. 

!  OS) _  Design:  Deciding  upon  methodology,  selection  of  techniques  of  data  collection 

and/or  analysis,  design  of  data  collection  instruments,  etc. 

i  AS) _  Analysis:  Description,  explanation,  interpretation  of  data. 

!  FSi _ Formulation  of  Recommendations:  Recommendation  of  adoption  of  new  or 

revised  programs,  policies,  procedures,  regulation,  manuals,  equipment,  courses 
of  instruction,  legislation,  etc. 


Indicate  your  estimate  of  the  user  organizational  involvement  by  circling  the  appropriate  scale 
value  below 

fpfc)  For  this  research  project: 

1  2  3  4  5  6 


The  total  amount  There  should  have 

was  sufficient  been  more 


<S8)  Were  you  personally  involved  in  any  of  the  above  stages  of  this  research  project? 

_ Yes  _ No 


A-3 


<  S9)  Comments: 


V.  HOW  SATISFIED  ARE  YOU  WITH  EACH  OF  THE  FOLLOWING  ASPECTS  OF  THIS 
RESEARCH? 

(Fill  in  the  blank  with  one  of  the  number  below:) 

1.  Very  satisfied  4.  Somewhat  dissatisfied 

2.  Somewhat  satisfied  5.  Very  dissatisfied 

3.  Neither  satisfied  or  dissatisfied 

f  ST  I  _  Timeliness  of  response? 

i  SC  i  _  Completeness  of  study? 

(SRI  _  Clarity  of  recommendations? 

i  $F )  _  Relevance  of  findings  to  the  problem? 

(SI)  _  Feasibility  of  implementation? 

ISP)  _  Cost  of  implementation? 

SB*  _  Projected  benefits  of  implementation? 

(SO)  PROVIDE  ANY  INFORMATION  AVAILABLE  ON  THE  SATISFACTION  OR  DISSATIS¬ 
FACTION  OF  THE  USER  WITH  THE  RESULTS  OF  THE  RESEARCH. 

(Provide  documentation  if  available:  e.g.,  letters,  memorandums,  etc.) 


VI.  HAS  THERE  BEEN  A  FORMAL  OR  MANAGEMENT  REVIEW  OF  THE  IMPLEMENTATION 

POTENTIAL  OF  THIS  REPORT,  E.G.,  BY  DECISION-MAKING  INDIVIDUALS  OR  COMMITTEES 
REPRESENTING  YOUR  COMMAND  OR  HIGHER  ECHELONS? 

(FF)  Yes _ No _ 

If  yes,  please  describe  the  findings  of  this  review.  If  no,  why  not?  Please  indicate  below  if  a 

decision  has  been  reached  to  definitely  implement  the  product 


A- 4 


VII.  THE  RESEARCH  WILL  IMPACT  AS  FOLLOWS  (IF  APPLICABLE): 
Place  a  "P"  next  to  each  item  the  research  has  a  potential  impact  on. 

Place  an  “A"  next  to  each  item  that  the  research  has  already  impacted  on. 


GO; 

•OP; 

•  Ol  f 
IOM? 


Operational  Commanders 
Changes  in  doctrine 
Changes  in  procedures 

Information  on  human  capabilities  and  limitations 
Modification  in  requirements  for  manpower  or  equipment 


t'CPI 
>:  CC ; 
«:CR; 
ft;  Si 
•CD; 

(DH 
-OR; 
(DM  I 
•'.Vi  P) 

•  1  Ds 
■ )  n> 

(  i  M ; 


■  SHI 
(Dti 
(HP; 
•HE> 
•DP: 


JiH; 
iRM; 
i  R 1 ) 
'  R  R  !■ 


Personnel  and  Manpower 

Changes  in  management  policy  or  techniques 

Changes  in  planning  capability 

Changes  in  manpower  requirements 

Solutions  to  specific  problems 

Information  on  which  to  base  R&D  requirements 

Training  Managers 

Development  of,  or  change,  in  course  of  instruction  or  training  programs 
Development  of  requirements  for  training  curricula  and  equipment 
Development  of,  or  change  in,  instructional  delivery  methods  and  media 
Changes  in  management  policy  or  practices 
Training  device  prototypes 

Information  on  which  to  base  long  range  objectives  and  further  R&D  requirements 
Evaluation  of  specific  materials  and  procedures 

System  Developers 

Information  on  human  capabilities  and  limitations 

Evaluation  of  specific  designs 

Human  Factors  Engineering  (HFE)  design  principles 

Efficient  ways  of  applying  HFE 

Changes  in  development  management  practices 

R&D  Community 

Information  about  human  capabilities  and  limitations 

Information  on  the  effectiveness  of  various  manpower,  personnel,  or  training  programs 
Solutions  to  technical  problems 
Identification  of  further  R&D  requirements 

Other  (describe) 
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VIII.  WILL  THE  POTENTIAL  OR  ACTUAL  IMPACT  OF  THE  RESEARCH  RESULT  IN  EITHER  COST 
W8?  SAVINGS,  OR  INCREASED  EFFICIENCY  OR  EFFECTIVENESS? 

_ Yes  _ No 

If  yes,  explain  how. _ _ 


IX.  DID  THE  RESEARCH  SATISFY  THE  USER  NEEDS?  (Circle  one  of  the  responses.) 

1.  To  a  very  great  extent  4.  To  a  little  extent 

2.  To  a  great  extent  5.  To  a  very  little  extent 

3.  To  a  moderate  extent  6.  Not  at  all 

Explain  or  expand  your  response  to  the  question  above  if  applicable. 


X.  WHERE  DO  YOU  BELIEVE  THE  AUTHORITY  TO  IMPLEMENT  THE  BULK  OF  THE 
RECOMMENDATIONS  (IF  ANY)  IS  LOCATED? 

A.  _  The  operational  unit  level 

B.  _ The  intermediate  management  level 

C.  _  The  major  organization  management  level  (Fleet  or  Systems  Command) 

D.  _  Office  of  CNO/Chief  of  Staff 

E.  _  Secretary  of  Defense  management  level  or  above 

F.  _  No  recommendations  were  made 

XI.  WHICH  OF  THE  FOLLOWING  BEST  DESCRIBES  THE  ROLE  OF  YOUR  ORGANIZATION 
CONCERNING  THE  IMPLEMENTATION  OF  THE  RECOMMENDATIONS? 

A _ We  have  no  role  in  implementation 

B.  _  We  have  only  an  advisory  role 

C.  _  We  participate  in  a  group,  committee  or  council  which  decides 

D.  _ We  have  the  authority  to  make  the  implementation  decision 

E.  _  No  recommendations  were  made 

XII.  IF  YOUR  RESPONSE  TO  THE  ABOVE  QUESTION  WAS  "NO  ROLE",  OR  "ONLY  AN  ADVISORY 
■  ROLE",  WHO  DOES  HAVE  THE  AUTHORITY  TO  MAKE  IMPLEMENTATION  DECISIONS? 


XIII.  IF  DOLLAR  RESOURCES  ARE  REQUIRED  FOR  IMPLEMENTATION  AND  OPERATIONAL 
,  x7)  USE,  HAVE  THEY  BEEN  IDENTIFIED  AND  ADDED  TO  THE  POM  SUBMISSION? 

_ Yes  _ No  _ Not  required 


XIV.  COMMENTS?: 

i  U  T  ■ 


THANK  YOU  FOR  PROVIDING  THIS  EVALUATION 
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